The computer has proven to be one of the most remarkable discoveries in human civilization and has been exploited to the greater good of mankind in the last half of the twentieth century. The use of computers in information storage, data processing, automation, computation, and other applications has greatly simplified many existing tasks while at the same time expanding the range of tasks that can be accomplished by humans. Many complex or laborious tasks can be automated, made more efficient, and completed more rapidly through the use of computers.
Today's computers are characterized by two fundamental attributes. First, computers operate with binary logic. The storage and manipulation of data occurs through conversion to binary strings and transformation of binary strings according to a particular computational objective. The requirement for binary processing is a consequence of the two state nature of silicon, the material that functions as the working substance of today's computer. Binary 0 and 1 states can be readily established in silicon and manipulated to perform computations.
Second, today's computers operate in a sequential fashion. Completion of a computational function is inherently a step by step process. Computer programs are simply line by line instructions that outline a sequence of steps to be implemented by a computer. The steps are executed one by one with the result of preceding steps typically being forwarded to later steps to effect a computation.
Despite their tremendous successes, certain computations or functions remain largely unamenable to solution or implementation by modem computers. Examples of such computations or functions include pattern classification, pattern association, associative memory functions, speech, and character recognition. In fact, many of the tasks that are difficult or cumbersome to implement with a conventional computer are tasks that are readily and intuitively performed by humans and other biological organisms. The recognition of a familiar face in a crowd, for example, is a task that even a small child can perform and yet remains a difficult task for a conventional computer.
The realization of the difficulties in implementing certain routine human tasks with a conventional computer has motivated much effort at better understanding human thought processes and developing computing schemes and devices capable of mimicking, emulating or at least approximating biological information storage and processing. These efforts have spawned the field of neural networks and reinforced appreciation of the unique, largely non-sequential nature of human thought.
Primitive versions of neural networks first came into existence more than 50 years ago with pioneering work by Warren McCulloch and Walter Pitts, on the one hand and Donald Hebb on the other hand. Their simple models were extended and refined in the 1950's and 1960's in work that led to neural networks such as the perceptron, ADALINE and MADALINE. The 1970's were a quiet period for neural networks as researchers began to realize that implementation of neural networks was more difficult than expected. Since about 1980, however, activity in neural networks has exploded as more sophisticated networks, better modeling schemes, and more robust computing methods have been developed. Advances such as the back-propagation method, adaptive resonance theory, self-organization, competitive learning models, multilayer structures and the neocognitron have vastly improved neural networks and provided the impetus for significant continuing activities.
One goal of neural network research is to develop systems that function like the human brain. The neuron is the basic learning unit in the brain of a human. A schematic depiction of a neuron is presented in FIG. 1 herein. A neuron functions by receiving a weighted input signal from its dendrites and responding according to the level of the signal. If the input signal is below some threshold characteristic of the neuron, the neuron does not respond to the signal. If the input signal exceeds the threshold, the neuron responds by transmitting an action potential signal along its axons to the axonic endings. In general, a neuron may have a large number of dendritic inputs and a large number of axons.
The brain comprises billions of neurons assembled in a complex interconnected arrangement. The action potentials at the axonic endings of one neuron provide inputs to succeeding neurons. A signal determined by the action potential is transmitted from one neuron to another at the synapse. The synapse is a junction of the axonic ending or pre-synaptic terminal of one neuron with a dendrite or post-synaptic terminal of another neuron as shown in FIG. 2 herein. The action potential induces the release of neurotransmitters from the pre-synaptic terminal across the synaptic cleft to the post-synaptic terminal of the receiving dendrite. The neurotransmitters enter the post-synaptic terminal and react with proteins therein to produce an electrical potential or charge at the membrane of the post-synaptic terminal. The magnitude of this charge is determined by the strength of the incoming signal (e.g. the number or type of neurotransmitters) and a weighting factor characteristic of the synapse, its function and history. The potential after weighting constitutes an input signal to the dendrite. The weighted signal at each dendrite may be excitatory or inhibitory. The neuron senses the net combined weighted inputs from all of its dendrites in a given interval of time and if this combined input exceeds the threshold of the neuron, it fires, transmits an action potential to its axonic endings for introduction to the dendritic inputs of the next layer of neurons and the process repeats.
In the brain, each neuron is synaptically connected to about 1000 other neurons and when a neuron fires, it sends an action potential to some or all of the neurons to which it is connected. Each of these neurons in turn is connected to another ca. 1000 neurons and a cascading weighted-threshold activation scheme over many layers of neurons occurs. As a result, the neurons in the brain are highly interconnected and form a massively parallel network for signal processing. The higher order reasoning skills of humans and other biological organisms, including the ability to learn and adapt, are thought to be due to the high degree of parallelness and interconnectivity of neurons.
Neural network systems include a plurality of interconnected nodes where each node is intended to incorporate several basic aspects of biological neurons. First, neural network nodes generally include the ability to receive multiple input signals. Second, neural network nodes generally include the concept of an activation threshold for firing with the threshold controlling whether the node fires (i.e. transmits a signal to one or more succeeding nodes). If the cumulative signal received by a node meets or exceeds the threshold, the node fires. If not, the node does not fire and no signal is transmitted. The signal produced upon firing may be referred to as an activation signal. Sometimes this activation signal is the output signal of the node and is directly provided to succeeding nodes, but frequently the activation signal is modified according to an activation or transformation function to provide the output signal of the node. The output signal of a node becomes an input signal to succeeding interconnected nodes. Third, neural network nodes generally include weighting of signals. The signals received by a node from preceding nodes are weighted versions of the output signals produced by the preceding nodes. The weighting is most often achieved by multiplying the output signal provided by a preceding node by a weighting factor. The weighted signals from all interconnected preceding nodes are combined and compared to the threshold to determine whether or not the signal is further propagated to later nodes.
In order to best achieve the learning and adaptability capabilities of biological systems, it is desirable for the nodes in a neural network to approximate the function and behavior of biological neurons as closely as possible. An intrinsic property of biological neurons is their ability to accumulate signals from a large number of inputs and instantly initiate firing upon attainment of a threshold condition. Currently, neural networks are implemented with conventional binary processing computers based on silicon as the computing medium. In order to test whether the activation threshold is met, for example, a series of classic addition and comparison operations must be executed and if warranted, an output signal is generated. The use of silicon as a computing medium does not permit a neural network node to intrinsically or innately respond to the plurality of input signals that it receives. Instead, a series of offline calculations is completed that at best only approximates or simulates the response of a biological neuron. Also, inputs from separate sources are typically processed by independent memory or processing units rather than centrally at a single unit as is the case with a biological neuron. The sequential nature of conventional computing limits the fidelity of neural networks by requiring the separate and serial processing of information. A computing medium that better mimics the innate behavior of a biological neuron is needed to improve the functionality of neural networks.